Voice operated control device

ABSTRACT

A control device may transmit electrical signals to a wireless hub to control of a display device, in a first zone, registered to the wireless hub. The control device may receive, from the wireless hub based on a user initiated source change, a control signal to assign buttons for control of the display device. The control device may transmit to the wireless hub a first signal, based on a voice command, to control content and a second signal, based on a gesture command, for a source change on the display device.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/214,111 entitled Voice Operated Remote Control, filed Jul. 19, 2016,which issued as U.S. Pat. No. 10,523,740 on Dec. 31, 2019, which is acontinuation of U.S. patent application Ser. No. 14/044,288 entitledControl of Data Presentation in Multiple Zones Using a Wireless HomeEntertainment Hub, filed Oct. 2, 2013, which issued as U.S. Pat. No.9,398,076 on Jul. 19, 2016, which is a continuation of U.S. patentapplication Ser. No. 11/563,503 entitled Control of Data Presentation inMultiple Zones Using a Wireless Home Entertainment Hub, filed Nov. 27,2006, which issued as U.S. Pat. No. 8,607,281 on Dec. 10, 2013, which isa continuation-in-part of U.S. patent application Ser. No. 11/470,862,entitled Data Presentation Using a Wireless Home Entertainment Hub,filed Sep. 7, 2006, which issued as U.S. Pat. No. 8,935,733 on Jan. 13,2015, the entire disclosures of which are incorporated herein byreference as if fully set forth.

This application is related to co-pending U.S. patent application Ser.No. 11/535,211, entitled Device Registration using a Wireless HomeEntertainment Hub, filed Sep. 26, 2006; patent application Ser. No.11/535,216, entitled User Directed Device Registration using a WirelessHome Entertainment Hub, filed Sep. 26, 2006; U.S. patent applicationSer. No. 11/535,232, entitled Source Device Change using a Wireless HomeEntertainment Hub, filed Sep. 26, 2006; U.S. patent application Ser. No.11/470,872, entitled Control of Data Presentation using a Wireless HomeEntertainment Hub, filed Sep. 7, 2006; U.S. patent application Ser. No.11/470,879, entitled Data Presentation from Multiple Sources using aWireless Home Entertainment Hub, filed Sep. 7, 2006; U.S. applicationSer. No. 11/470,895, entitled Control of Data Presentation from MultipleSources using a Wireless Home Entertainment Hub, filed Sep. 7, 2006,U.S. application Ser. No. 11/563,366, entitled Control of Access to DataUsing a Wireless Home Entertainment Hub, filed Nov. 27, 2006; U.S.application Ser. No. 11/563,486, entitled Inventory of HomeEntertainment System Devices Using a Wireless Home Entertainment Hub,filed Nov. 27, 2006; U.S. application Ser. No. 11/563,520, entitledConnecting a Legacy Device into a Home Entertainment System Using aWireless Home Entertainment Hub, filed Nov. 27, 2006, and U.S.application Ser. No. 11/563,530, entitled Data Presentation in MultipleZones Using a Wireless Home Entertainment Hub, filed Nov. 27, 2006, theentire disclosures of which are incorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description will be better understood when readin conjunction with the appended drawings, in which there is shown oneor more of the multiple embodiments of the present invention. It shouldbe understood, however, that the various embodiments of the presentinvention are not limited to the precise arrangements andinstrumentalities shown in the drawings.

In the Drawings:

FIG. 1 is a system diagram of a home entertainment system according toone embodiment of the present invention;

FIG. 2 is a use-case diagram of a wireless home entertainment hub inaccordance with the home entertainment system of FIG. 1;

FIG. 3 is a sequence diagram of user-initiated automatic registration inaccordance with the home entertainment system of FIG. 1;

FIG. 4 is a sequence diagram of manual device registration in accordancewith the home entertainment system of FIG. 1;

FIG. 5 is a sequence diagram of source activation in accordance with thehome entertainment system of FIG. 1;

FIG. 6 is a sequence diagram of direct source to sink transmission ofdata within the home entertainment system of FIG. 1;

FIG. 7 is a sequence diagram of transmission of data directed by thewireless home entertainment hub within the home entertainment system ofFIG. 1;

FIG. 8 is a sequence diagram of operation of a remote control inaccordance with the wireless home entertainment system of FIG. 1;

FIG. 9 is a system diagram of an audio/visual receiver used to connectspeakers to the wireless home entertainment hub of FIG. 1;

FIG. 10 is a system diagram of a wireless network interface box used toconnect non-wireless enabled devices to the wireless home entertainmenthub of FIG. 1;

FIG. 11 is a system diagram of a multiple display device configurationin accordance with the home entertainment system of FIG. 1;

FIG. 12 is a system diagram of an implementation of using multipledisplay devices for a wide-angle display in accordance with the homeentertainment system of FIG. 1;

FIG. 13 illustrates a series of exemplary user interface display devicescreens for registering devices in accordance with the homeentertainment system of FIG. 1;

FIG. 14 illustrates a series of exemplary user interface display devicescreens for registering speakers in accordance with the homeentertainment system of FIG. 1; and

FIG. 15 illustrates a series of exemplary user interface display devicescreens for showing missing devices in accordance with the homeentertainment system of FIG. 1.

DETAILED DESCRIPTION

Certain terminology is used herein for convenience only and is not to betaken as a limitation on the embodiments of the present invention. Inthe drawings, the same reference letters are employed for designatingthe same elements throughout the several figures.

The words “right”, “left”, “lower” and “upper” designate directions inthe drawings to which reference is made. The words “inwardly” and“outwardly” refer to directions toward and away from, respectively, thegeometric center of the home entertainment system and designated partsthereof. The terminology includes the words above specificallymentioned, derivatives thereof and words of similar import.

Unified Modeling Language (“UML”) can be used to model and/or describemethods and systems and provide the basis for better understanding theirfunctionality and internal operation as well as describing interfaceswith external components, systems and people using standardizednotation. When used herein, UML diagrams including, but not limited to,use case diagrams, class diagrams and activity diagrams, are meant toserve as an aid in describing the embodiments of the present invention,but do not constrain implementation thereof to any particular hardwareor software embodiments. Unless otherwise noted, the notation used withrespect to the UML diagrams contained herein is consistent with the UML2.0 specification or variants thereof and is understood by those skilledin the art.

An exemplary home entertainment system (HES) 100 including wirelesslyconnected devices in accordance with one embodiment of the presentinvention is shown in FIG. 1. Interactions between the various wirelessdevices in the HES 100 and a user 120 are coordinated by a wireless homeentertainment hub (WHEH) 102. It is understood by those skilled in theart that a wireless device in the HES 100 may contain an external wirefor the purpose of supplying power to the wireless device.

Referring generally to FIGS. 1 and 2, devices in the HES 100 can broadlybe classified into two categories: source devices 122 and sink devices124. Source devices 122 transmit data within the HES 100. Source devices122 include, but are not limited to, DVD players 104, digital videorecorders (DVR) (not shown), set-top boxes (STB) 106 (e.g., cable orsatellite channel tuners), gaming consoles 108 (e.g. Xbox®,PlayStation®), CD players or other audio playback devices (e.g., MP3player) (not shown). It is understood by those skilled in the art thatexternal data can be introduced into the HES 100 for transmission by oneor more of the source devices 122 by various means, such as opticalfiber, co-axial cable, or a satellite dish system connected to the STB106. Sink devices 124 receive the transmitted data within the HES 100,sometimes converting a signal into an audible or visible stimulus. Sinkdevices 124 include, but are not limited to, speakers 110, audio/visualreceivers (AVR) 145 (see FIG. 9), and display devices 112 such as anHDTV or other television, monitor, or display screen or mechanism.

Those skilled in the art will recognize that a PC 114 can act as asource device 122 and/or a sink device 124 within the HES 100. The PC114 can act as an audio and/or a video source transmitting data from,for example, a CD, DVD, stored music and video media, or data accessedfrom the Internet. The PC 114 can also act as a sink device 124displaying video and audio data from, for example, the DVD player 104 orSTB 106.

The HES 100 may also utilize a remote control 118 and a calibrationdevice 116, discussed in greater detail below.

The WHEH 102 facilitates the transfer of data between the source andsink devices in the HES 100, as well as coordinates the interactionbetween the user 120 and the source and sink devices 122, 124. Forexample, referring to FIG. 2, the WHEH 102 may perform a registerdevices use-case, change source use-case, direct audio and video signaluse-case, program remote use-case, control volume use-case, andcalibrate system use-case, which are described in more detail below.Within the HES 100, the WHEH 102 controls the flow of data, informationand other “traffic” by recognizing the devices within the HES 100,tracking their current status (e.g., active, standby, etc.), directingthe transfer of data between devices, etc. In addition, the WHEH 102provides a central controller for the HES 100 that allows a user 120 tooperate the HES 100 in an efficient manner through interaction with theWHEH 102, which then subsequently provides instructions to the otherdevices in the HES 100 to perform the function requested by the user120. Such interactions with the WHEH 102 by the user may be performedthrough with a visual user interface presented on the screen of thedisplay device 112. Alternately, the remote control 118 may include adisplay screen, such as an LCD, to present the user with a visualinterface to the WHEH 102.

The WHEH 102 includes one or more wireless transceiver units tofacilitate the wireless transfer of data between the source and sinkdevices 122, 124 using wireless communication standards described below,a memory for storing data and other information generally used in theoperation of the HES 100, and a processor for executing the instructionsets for the functions of performed by the WHEH 102, including theuse-cases listed above. The WHEH 102 may exist as a standalone unit orit may be integrated into another device. For example, the WHEH may beincluded in the display device 112 or the remote control 118. Oneskilled in the art will recognize that the WHEH 102 can act as a sourcedevice 122 and/or a sink device 124 in the HES 100. For example, theWHEH 102 may receive data (i.e., acts as a sink unit) from a source unitcurrently transmitting data in the HES 100, and process and transmitthat data (i.e., acts as a source unit) to other sink devices in the HES100 for presentation to a user 120.

Ultra-wide band technology (UWB) utilizing orthogonal frequency divisionmultiplexing (OFDM) or a direct sequence communication system may beused for the wireless communication between the WHEH 102 and the sourceand sink devices 122, 124 in the HES 100. Those skilled in the art willrecognize that a number of other wireless commutation standards,including Bluetooth and various WiFi standards, can be used withoutdeparting from the spirit and scope of multiple embodiments of thepresent invention for transfer of data between devices within the HES100. In one embodiment, more than one wireless standard may be utilizedwithin the HES 100 to facilitate data transfer between the devices. Forexample, the WHEH 102 and source and sink devices 122, 124 may eachcontain a UWB transceiver for transfer of audio and/or video data and aWiFi transceiver for transferring operation instructions.

Referring generally to FIGS. 2-4, audio and video devices in the HES 100can be registered with the WHEH 102. The registration creates a uniqueassociation between devices and the WHEH 102 such that registereddevices belonging to the HES 100 are controlled by WHEH 102, and cannotsimultaneously be registered with a different home entertainment systemor another wireless home entertainment hub operating in a nearbyproximity, such as in a neighboring apartment or household.

User-Initiated Automatic Registration

Referring to FIG. 3, the user 120 initiates the registration procedure.The WHEH 102 sends a request for any unregistered device to respond. TheWHEH 102 request may include a unique identifier for the WHEH 102, suchas a WHEH ID number. A device response includes a unique deviceidentifier, such as a device ID number. Referring to FIG. 13, a list ofresponding devices 160 is presented to the user 120, and the user 120selects the device that is to be registered with WHEH 102. The WHEH 102sends a registration instruction to the selected device and theregistration is stored on both the device and the WHEH 102. Once aregistration has been established 162, the device is removed from thelist 164, and the registration process is repeated for the remainingunregistered devices within the HES 100 that have responded to the WHEH102.

In an alternate embodiment, the registration procedure is accomplishedautomatically between the WHEH 102 and unregistered devices. Forexample, the WHEH 102 may periodically broadcast a request for anyunregistered devices to report. In addition to replying with the deviceID number, an unregistered device can respond with a list of itscapabilities so that the WHEH 102 can determine the functionality of thedevice (i.e., if it is a DVD player, DVR, STB, etc.) before sending aregistration instruction. Alternately, the unregistered device canrespond with its make and model number from which the WHEH 102 candetermine the functionality of the device using an internal database ofdevices and also obtain any necessary command codes for the device,

Alternatively, the user 120 may initiate the broadcast for unregisteredsources instead of having the WHEH 102 perform a periodic broadcast forunregistered sources. For example, when a user 120 adds a new componentor device to the HES 100, a request to find unregistered sources may beinitiated, such that once the request is initiated, the remainingregistration procedure proceeds automatically as discussed above.

In an alternate embodiment, the WHEH 102 may automatically recognize andregister all devices in the HES 100. For example, a user 120 maypurchase a set of coordinated devices for wireless HES 100 including,for example, a display, set of multi-channel speakers, a DVD player 104,and a WHEH 102 (which may be a discreet device or contained in one ofthe system devices, such as the display or remote control). Each ofthese coordinated devices may contain a coordinated ID that isrecognizable by the WHEH 102. Additionally, the speakers may be labeledwith their intended position within the HES 100 (e.g., front left,middle right) to aid the user 120 in their correct placement. Uponplacement and power-up, without any additional actions by the user 120,the WHEH 102 automatically registers the coordinated devices based ontheir coordinated ID's that have been set by the device manufacturer.

The wireless HES 100 may perform an error checking during theregistration of the source and sink devices to make sure that the devicebeing registered matches the type of device being requested forregistration. The WHEH 102 can compare the list of capabilities receivedfrom the device during the registration with a list of expectedcapabilities stored in the WHEH 102. If the capabilities in the devicereply match the expected capabilities of the WHEH 102, an indication ofthe registration is stored in the WHEH 102 and the device. If thecapabilities and expected capabilities do not match, the registration isnot stored and may be re-initiated.

Manual Registration

The registration may also be performed manually by the user 120 (seeFIG. 4). The user 120 initiates a registration procedure by pressing aregistration actuator on the WHEH 102. Examples of actuators includebuttons, touch pads, touch screens, or any other actuating assemblyrecognized by those skilled in the art. The user 120 presses aregistration actuator on a selected source unit which sends a signal tothe WHEH 102 that a registration should be stored with this unit. If theWHEH 102 is unable to determine the functionality of the source (e.g.,DVR, DVD, etc.), the user 120 may manually assign the functionality ofthe source to complete the registration. For example, if the user 120selects the registration actuators on the WHEH 102 and the DVD player104, the WHEH 102 may cause “DVD registered” to be displayed if theselected source is recognized as a DVD player 104 by the WHEH 102. Ifthe selected source is not recognized, the WHEH 102 may prompt the user120 to select the type of source device from a list. Thus, in this case,the user 120 may select “DVD” in order to complete the registration.This process is repeated until all the unregistered sources have beenregistered with the WHEH 102, or similarly if a new source is added intoan existing system. In other embodiments, the user 120 may initiateregistration from a source device 122, a sink device 124, a remotecontrol 118, or over a network.

Multi-Instance Device Registration

For device types where multiple instances of the device exist within thesystem 100 (e.g., speakers 110), a number of approaches can be used toidentify each device's specific role. For speakers 110, the role of eachspeaker can be pre-identified by the manufacturer (e.g., “front-right”,“subwoofer”, etc.). The user-initiated automatic registration proceduredescribed above could be used to register the speakers 110 with the WHEH102 since the speaker 110 could identify itself, for example, as thefront left speaker, during the registration process. Alternatively, eachspeaker 110 could have a physical input that the user 120 could set toindicate the speaker's role (e.g. “front-left”, “back-right-surround”).In another embodiment, the WHEH 102 could use one or more microphoneswithin the HES 100 to elicit position and frequency responseinformation, or the HES 100 could use other position detectiontechnologies understood by those skilled in the art.

In another embodiment, each speaker 110 could have a registrationactuator to be activated in response to a WHEH 102 prompt for a speakerplaying a specific role. For instance, the WHEH 102 could prompt theuser 120 for the front-left speaker and the user 120 could activate theregistration actuator. Alternately, the user 120 may initiate theregistration procedure by activating a registration actuator on the WHEH102. The user 120 then presses a registration actuator on a speaker 110and identifies the functionality of that speaker 110 within the audiosystem 100. For example, at the time of registration, the user 120identifies the selected speaker as the left front, the repeats for rightfront, continuing until all the speakers 110 have been identified andregistered. In one embodiment, the WHEH 102 may prompt the user 120 witha list or graphic display of speaker positions available as shown inFIG. 14. The user 120 first selects the speaker 110 to be registered andthen presses the registration actuator on the selected speaker. In analternate embodiment, the WHEH 102 may first prompt the user 120 toenter the number of speakers to be registered with the WHEH 102 and thenselect the appropriate speaker configuration to match. For example, ifonly four speakers 110 are selected, the WHEH 102 would not present theuser 120 with a Dolby® Digital 7.1 speaker configuration, but a fourspeaker list of left and right front, and left and right rear.

After device registration is complete, the WHEH 102 may compare the listof source devices 122 and sink devices 124 registered with the WHEH 102to a list of possible types of source devices 122 and sink devices 124that can be registered with the WHEH 102. Using the display device 112,the WHEH 102 may present to the user 120 a list of device types that aremissing from the HES 100. The user 120 can indicate whether one or moreof the listed device types are present in the HES 100, indicative of anerror in the registration procedure. These devices can then beregistered with the WHEH 102 using any appropriate method describedabove. For example, after registering all the detected devices in a HES100, the WHEH 102 determines that a gaming console 108, a DVR, and asubwoofer are missing device types within the HES 100. A list of thesemissing device types is displayed on the HDTV. The user 120 inputs thatthe subwoofer is present in the HES 100. After the subwoofer issuccessfully registered, the user 120 is presented with a list of gamingconsole 108 and DVR as the missing device types. Referring to FIG. 15,in one embodiment, the WHEH 102 may present the user with a list ofdevices missing in the HES 100 based on requirements to fully supportplayback of program content. For example, if a DVD program contains a 5channel audio track, but only two speakers, front left and front right,are registered with the WHEH 102, the WHEH 102 may display an indicationto the user 120 that a rear left, rear right, center channel andsubwoofer may be added to the HES 100 to enhance the audio experience ofthe user 120.

Once the WHEH 102 has determined a list of missing device types, theWHEH 102 may then cause advertisements for the missing devices to bedisplayed to the user 120 on the display device 112. The advertisementsmay be generic advertisements for the missing device type or may besponsored advertisements for a specific brand of the missing device.Advertisements may be stored on the WHEH 102 or received fromprogramming channels accessible using the STB 106. Alternately, theadvertisements may be retrieved from a computer network (e.g. theInternet) through a direct connection of the WHEH 102 to the computernetwork or via a PC 114 connected to the computer network and registeredwith the WHEH 102. For example, the WHEH 102 determines that no DVR isregistered with the WHEH 102. The WHEH 102 transmits an advertisementfor a DVR stored in the WHEH 102 to the display device 112 forpresentation to the user 120 just after the user 120 has initiated theentertainment session and before displaying the requested programming.Alternately, the WHEH 102 may insert advertisements into the programmingby replacing an advertisement from the programming stream with anadvertisement for the missing device or device type. For example, if agaming console 108 is determined to be missing, the WHEH 102 may detectan advertisement for a gaming console 108 on a programming channelreceived on the STB 106. The WHEH 102 stores the advertisement for thegaming console 108 and replaces advertisements in the regularprogramming stream with the stored advertisement for a gaming console108.

The presentation of the advertisement may be repeated for apredetermined length of time (e.g. for 4 weeks) or until the missingdevice is registered with the WHEH 102. The insertion of advertisementsfor missing devices or device types may also be limited to a range ofdates and/or times. For example, advertisements for gaming consoles 108may be presented to the user 120 from the middle of November until theend of December to correspond to a holiday shopping season. Alternately,the insertion of the advertisements may be based on an identification ofthe user by the WHEH 102.

Source Selection

Referring generally to FIGS. 2, 5-7, in addition to coordinating theregistration of device within the HES 100 as described above, the WHEH102 is also used to coordinate and/or control the state of the sourceand sink devices and the transfer of data from the source devices tosink devices during typical operation of the HES 100. Device states(also referred to as modes) may include “on”, “off”, “active”, “lowpower”, “standby”, etc. Data may include instructions, audio/videoprogramming, or any other information generally passed between or amongsource/sink devices. Some examples of typical operations that are commonin the general utilization of the HES 100 by the user 120 are a requestor action by the user 120 to activate a source (e.g., start watchingprogramming from a cable broadcast) or initiate a change from one sourcedevice to another (e.g., discontinue watching programming from a cablebroadcast to watching a movie on the DVD player). The request toactivate a source device or to change from one source device to anothercan be accomplished in a several ways. The user may initiate the actionthough the use of the remote control 118, or interact directly with asource device. For example, when a user inserts a DVD into the DVDplayer 104, it automatically causes the WHEH 102 to activate the DVDplayer 104 (or initiate a source change as described below if anothersource is already active in the HES 100). In either case, once therequest has been made by the user 120, the WHEH 102 completes theprocess as described below.

FIG. 5. is sequence diagram showing the selection of a source device 122by the WHEH 102 in one embodiment of the HES 100. When a source deviceis activated, it begins transmitting data to the HES 100. Theinstruction to activate also causes an internal reference count withinthe source device to increment by one, where the reference countrepresents the current number of zones (described in more detail below)that are receiving data from the source device For example, if thesource device, currently in standby mode, is activated, its referencecount increases to one. After a user 120 initiates a request to changeto a new source, the WHEH 102 instructs the current active source deviceto decrement its internal reference count by one. When the active sourcedevice internal reference count is zero, the source device may stoptransmitting and enter a low power or stand-by mode. If the internalreference count is not zero, the source device continues to transmitsince there are other devices still listening to its transmission. TheWHEH 102 then instructs the newly selected source unit to activate,including increasing its internal reference count by one, and the newlyselected source device begins transmitting data to the HES 100. The sinkdevices may receive the transmitted data directly from the currentactive source or from the WHEH 100, both described below. In analternate embodiment, a list of the sink devices 124 in one or morezones of the HES 100 that are listening for data from the source device122 is stored in the source device 122. Sink devices 124 are added orremoved from the list as directed by the WHEH 102. When there are nosink devices on the list, the source device may stop transmitting andenter a low power or stand-by mode.

Direct Source to Sink Data Transmission

Referring to FIG. 6, after a source activation or change is initiatedwithin the HES 100 as described above, the WHEH 102 broadcasts to allsink units, or those that are relevant, an instruction to discontinuereceiving and transmitting data from the previously active source andbegin receiving the transmitted data from the newly selected source,where the activities in FIG. 6. labeled “transmitVideo( )” and“transmitAudio( )” represent a continuous stream of data from the sourcedevice 122 to the sink devices 124. This instruction from the WHEH 102may be broadcast as a single instruction to all units (i.e., a commoninstruction recognizable by any device in the HES 100) or may be adistinct instruction sent to each of the sink units. Audio and/or videodata from the current active source device is transmitted directly tothe relevant sink devices as instructed by the WHEH 102. For example,the display device 112 and speaker 110 receive and present the videodata and audio data, respectively, from the current active sourcedevice. In one embodiment, the newly selected source device that hasbeen activated in the HES 100 may transmit one or more instructionsdirectly to the sink units to begin receiving and presenting the datafrom the newly selected source and discontinue presenting the data fromthe previously active source.

Source to Sink Data Transmission through the WHEH

In an alternate embodiment, the sink units in the HES 100 receive datafrom the wireless home entertainment hub (see FIG. 7). The WHEH 102receives the audio and video data from the current active source deviceand transmits the audio and video data to the appropriate sink unit. Ifa source change is initiated within the HES 100 as described above, thesink units may be unaware of a change of source with the HES 100 sincethey always listen to (i.e., receive data from) the WHEH 102, and notdirectly to the active source device.

In one embodiment, more than one source can be designed as an activesource by the WHEH 102. Data from multiple active sources can besimultaneously presented by the relevant sink devices as described bythe two methods above. The WHEH 102 receives the data from the two ormore active source devices and transmits the data to the relevant sinkdevices. The WHEH 102 may process (e.g., mix) the data from the two ormore source devices before transmitting. Alternately, the WHEH 102 mayinstruct the sink devices to listen to and present data transmitteddirectly from the two or more active sink devices.

Referring to FIG. 8, in one embodiment of the present invention, aremote control 118 is used with the HES 100. The remote control 118receives actuator assignments based on the currently active source inthe HES 100 from the WHEH 102. For example, if the DVD player 104 iscurrently the active source, the actuator assignment on the remotecontrol 118 is for the DVD player 104. When a user 120 activates anactuator on the remote control 118, the actuator selection is sentdirectly to the DVD player 104, which responds with the correspondingactivity for that actuator. If the user 120 initiates a source changethrough the WHEH 102 as described above, (e.g. from the DVD player 104to the set-top tuner), then the WHEH 102 sends a new actuator assignmentto the remote control 118 for the set-top tuner. Actuators activated onthe remote control 118 by a user 120 now cause an activity in theset-top box instead of the DVD player 104. This method of operation ofthe remote control 118 is referred to a “dumb” remote. The functionalityof the actuators on the remote control is controlled by the WHEH 102based on the current active source in the HES 100. The remote does notneed to store any information about past or present states of the HES100 or registration information between the devices in the HES 100.

In an alternate embodiment, a “smart” remote may be used in conjunctionwith the HES 100. The smart remote learns and stores the systemconfiguration, i.e., what source and sink devices are registered withthe WHEH 102. It also learns and stores the current state of the system,i.e., what sources and sinks are active. In addition, the smart remotestores the actuator assignments in an internal memory and may storesystem status information along with device registration information.When a user 120 requests a source change using the remote, the WHEH 102activates the new source as describe above, and the functionality of theremote control 118 is switched to controlling the newly active sourcewithout any input from the WHEH 102 to re-assign the actuators asdescribed in the dumb remote case above. In one embodiment, the WHEH 102may be contained in the smart remote instead of the display device 112.

A handheld mobile device, such as cellular phone or personal digitalassistant, can use appropriate wireless capabilities to communicate witha WHEH 102, obtain information to build and present a user interface,and serve as a remote control 118 for the HES 100. In addition, thecapabilities of the HES 100 may be used to enhance the functionality ofthe handheld mobile device. For example, when a cellular phone is activethe remote control 118, the active display device 112 may displayCallerID information or other information generally presented on thecellular phone display to the user 120 during an incoming telephonecall.

The WHEH 102 may respond to voice commands. A user 120 can perform someor all of the functionality of the remote control 118 by using simpleaudible commands. For example, to change the STB to channel 21, the user120 might say “Channel 21” and the WHEH 102 sends the correspondinginstruction to the STB to complete the channel change, or use thecommand “Volume Up” increase the system volume, where the WHEH 102 sendinstructions to the active sink unit to increase volume.

In one embodiment, the WHEH 102 may respond to physical gestures made bythe user 120 with the remote control 118. A user can provideinstructions to the WHEH 102 corresponding to a predetermined set ofphysical motions of the remote. The remote control 118 may include amotion sensing system that can relay motion information in up to3-dimensions to the WHEH 102. Additionally, the WHEH 102 or remotecontrol 118 may include directional sensors to determine the orientationof the remote control relative to the HES 100 or sense rotation. Suchmotion sensing systems and directional sensors are understood by thoseskilled in the art. For example, a user wishing to switch from watchingthe DVD player to the STB 106 may simply point the remote at the STB106, actuating an actuator on the remote control 118 to select aprogramming channel. Alternately, channel selection may be accomplishedby a quick series of left or right motions of the remote control, eachleft or right motion decrementing or incrementing, respectively, thechannel displayed by the STB 106. Volume control may be accomplished,for example, by a vertical motion of the remote control 118 directed atthe display device 112 or speakers 110. In this manner, the volume ofeach speaker can be adjusted independently of the other speakers. In oneembodiment, gestures may be used to move presentation of video contentfrom one display device to another, engage picture-in-picturefunctionality, or perform other manipulations.

Power Management

At startup of an entertainment session, a user 120 may direct a power-onmessage to the WHEH 102, via a remote control 118 or perhaps via anactuator on the WHEH 102 or device containing WHEH 102 functionality.The WHEH 102 can then retrieve input from the user 120 regarding thecapabilities required for the session and send activation messages toappropriate devices.

In another embodiment, a source device 122 can send a broadcast messageto the sink devices 124 in the home entertainment network indicatingrequired presentation capabilities. Required devices can activatethemselves and unneeded devices can enter a low-power state. Forexample, all devices in the system are in a lower power mode. The userinserts a DVD into the DVD player 104 which causes the DVD to becomeactive. The WHEH 102 detects the activity of the DVD player 104 andinstructs the display device 112 and speakers 110 to activate.Alternately, the display device 112 and speakers detect the activity ofthe DVD player 104 directly and activate.

In one embodiment, upon indication from the active source device of therequired audio output channels, the WHEH 102 sends signals to the audiosink devices to indicate whether or not they should remain active. Forinstance, upon indication of a Dolby® Digital 2.1 program, the WHEH 102may communicate to the side and rear speakers that they may enter a lowpower mode. Similarly, when an audio-only program is indicated, forexample from the CD player, the WHEH 102 can communicate to the videodisplay device 112 that it may enter a low power non-display state.

Devices in the HES 100 may contain low-powered radios (i.e.,transceivers) that poll for activity or constantly monitor for WHEH 102messages during a low-power device state. Wireless radios conforming tothe “ZigBee” standards can be used in some embodiments. “Bluetooth” or“WiFi” radios can be used in other embodiments. Modes of “UWB” can alsobe used to detect communications during low-power operations.

Volume Control

Volume control, including system muting, can be accomplished in multipleways. In one embodiment, all active audio sink devices may individuallydetect that the remote control 118 has transmitted an instruction tochange the volume. For instance, the eight speakers of a 7.1 surroundsound system each detect a “volume up” instruction transmitted from theremote control 118. Each speaker then increases the gain on its internalamplifier, thereby driving the speaker to higher volume. Alternatively,the WHEH 102 can detect an instruction from the remote control 118requesting a volume change and transmit to all the audio sink devicesone or more instructions to change their volume.

System Calibration

For calibration of the HES 100, a wireless calibration device 116 may beplaced at a typical viewing and/or listening position 121 (e.g., near oron a chair or couch that a user 120 would sit to view the television) ofthe HES 100 by the user 120 (see FIG. 1). For audio calibration, theWHEH 102 could direct a calibration signal to each audio device insequence or in combination, or each audio device could be directed togenerate its own calibration program. A microphone in the wirelesscalibration device 116 monitors the calibration signals from the audiodevices, and can communicate its readings to the WHEH 102 or back to theaudio devices in a point-to-point or broadcast mode. Adjustments couldthen be made to the frequency characteristics, volume, or otherparameters to provide a calibrated home theater environment. Similarly,a wireless light sensitive device could be used to monitor a calibrationsignal from one or more display devices 112 to provide video calibrationof the system. In one embodiment, the microphone and light sensitivedevice may be contained in a single calibration unit. Alternately, themicrophone and light sensitive may be contained in separate calibrationunits. In one embodiment, an actuator on the microphone device couldcause the device to communicate with the hub to initiate the calibrationsequence.

A/V Receiver

Referring to FIG. 9, one or more audio/visual receivers (AVR) 145 oramplifiers can be used to connect the HES 100 to speaker system. Thespeaker system may be wired 140, wireless 142 or a combination thereof.For example, the front speakers 142 in a four speaker system may bewirelessly connected 146 to the AVR 145, while the rear speakers 140 areconnected by a wire 148 to the AVR 145. The AVR 145 is registered withthe WHEH 102 and receives audio data from an active source (not shown)or through the WHEH 102, as described above. The AVR 145 transmits thereceived audio data to the appropriate speakers 140, 142, either throughthe wired connection 148 or by wireless communication 146. It should benoted that if the AVR 145 is configured for use with wireless speakers142, these wireless speakers 142 are not registered with the WHEH 102 asdescribed above and do not receive data from the WHEH 102 or sourcedevices 122 in the HES 100, but instead communicate with the AVR 145using methods understood by those skilled in the art.

Legacy Adaptor

Referring to FIG. 10, a home theatre network interface box (HTNIB) 125may be used to connect “legacy” devices 130 into the HES 100, includingthe WHEH 102. Legacy devices generally include those devices thatrequire hardwire connection for transmission and/or receipt of data andare not wireless enabled (e.g., an analog television connected to aset-top box using coaxial cable), although a wireless device may also beconsidered a legacy device if the wireless device cannot be configuredto communicate with the wireless home entertainment hub. The legacydevice is connected to the HTNIB 125 using a hardwire connection 128(e.g. coaxial cable). The HTNIB 125 is capable of being registered withthe WHEH 102, and recognized by the WHEH 102 as the legacy device towhich it is connected. The WHEH 102 directs data to and/or from theHTNIB 125 as appropriate to the type of legacy device to which the HTNIB125 is connected. The HTNIB 125 passes data to and/or from the connectedlegacy device as required by the current configuration of the HES 100.If a legacy source device outputs data in either an analog format or adigital format different than that used by the WHEH 102, the HTNIB 125can convert the output data into a digital format compatible withtransmission between the WHEH 102 and registered devices within the HES100. Similarly, if the HTNIB 125 is connected to a legacy sink device,the HTNIB may convert the digital data from the source device intoeither an analog format or a different digital format compatible withthe legacy device. For example, if a video cassette recorder (VCR) isconnected to the HTNIB 125, the WHEH 102 will recognize the NTNIB 125 asa VCR, and when the user 120 selects the source unit VCR, will instructthe sink devices 124 in the HES 100 to listen to the transmission fromthe NTNIB 125, which is transmitting the data received from the cableconnected to the VCR. One or more HTNIBs 125 could be used in the HES100 to connect one or more legacy devices. In one embodiment, a singleHTNIB 125 could be used to connect one or more legacy devices to the HES100, wherein the HTNIB 125 contains one or more connections for sink andsource devices. Each connection can be uniquely registered with the WHEH102.

Multi-Zone Operation

A single WHEH 102 may provide programming to multiple sets of sinkdevices that are registered with the WHEH 102. The HES 100 may bepartitioned into one or more zones. Each sink device 124 in the HES 100can be assigned to a zone. Zone assignment may be performed at the timeof device registration with the WHEH 102. Zone assignment or changingzone assignments can also be accomplished at any time after deviceregistration. An example of zone partitions within a HES 100 is thatzone 1 includes the display device and 7.1 speaker system in the livingroom; zone 2 includes a display device in the bedroom; zone 3 includesan AVR 145 connected to speakers in the kitchen; and zone 4 includes aPC 114 in the home office. Multi-zone operation allows users 120 indifferent partitions of the HES 100 to received data from differentsource devices 122 registered with the WHEH 102. For instance, the sinkdevices in zone 1 are presenting the program from an HD-DVD, while thespeakers in the kitchen connected to the AVR 145 in zone 3 arepresenting audio from a wireless music storage device that is alsoregistered with the WHEH 102. Zone assignments can be designed by theuser 120. Alternately, devices can be assigned to a zone in the HES 100by the WHEH 102 based on determining the location of the device andidentifying clusters of device as separate zones. The device locationsmay be input by the user during or after the time of deviceregistration, or the WHEH 102 may automatically determine the locationsof registered devices.

In one embodiment, the WHEH 102 can receive an audio source signalcontaining more channels than can be presented in the current HES 100(e.g., the audio signal is configured for a 7.1 system, but theinstalled HES 100 utilizes a 5.1 speaker configuration.) The WHEH 102can process and downmix the audio signal for presentation on theavailable speaker configuration. As described above, the WHEH 102 mayalso provide to the user 120 an indication that the audio signalcontains more audio channels than the current configuration of the HES100 can support, and recommend to the user 120 that additional speakers110 be added to the HES 100 to fully support playback of such audio.

The HES 100 may also contain more than one display device 112 assignedto the same zone. Each display device 112 is registered with the WHEH102 using one of the methods described above. The user may choose todesignate one of the display devices 112 as the primary display deviceduring or after registration of the display device 112, or alternatelythe WHEH 102 may automatically designate one of the display devices 112as a primary display based on the characteristics of the display device,such a screen size or pixel density, with the other display devicesbeing designated as secondary displays. For example, referring to FIG.11, if a 51″ HDTV 180 and two 32″ HDTV's 185 are registered with theWHEH 102 and assigned to the same zone, the WHEH 102 may automaticallydesignate the 51″ HDTV 180 as the primary display based on larger screensize, while the 32″ HDTV's 185 become the secondary displays. Secondarydisplays can be used to show supplemental program content that iscomplimentary to the main program content being displayed on the primarydisplay. This supplemental program content may be encoded in the mainprogram stream received from a content service provider, or may betransmitted as a separate program stream or side channel. Some examplesof supplemental program content that may be shown on the secondarydisplays include viewer e-mails during a talk show, stock prices,financial, or other information about a company during a business reportdirected to that company, and scores, playing schedules other teaminformation during a sportscast related to that team. The secondarydisplay may also present alternate views of an event during newsreporting. For example, the on-location reporter is on the primaryscreen, while alternate video related to that location is shown on thesecondary displays. The secondary display may also be used to displayextra program content included on a DVD. For example, bloopers ordirectors commentary corresponding to the scene of a movie presented onthe primary display can be shown on the secondary display.

The secondary display may also be used to present advertisementsconcurrent with main program content shown on the primary display. Theadvertisements may be related to a product currently being featured inthe main program content, such as for an automobile or a brand of foodor drink.

The secondary display can present to the user 120 a websitecorresponding to an Internet address displayed on the primary display.The Internet address may be transmitted along with but separate from theprogram content and received by the WHEH 102, which detects the Internetaddress in the program stream and retrieves the content of the websiteusing a network connection available within the HES 100. Alternately,the WHEH 102 may derive a web address shown on the primary displaythrough OCR on frames formed from the program content of the maindisplay, or the WHEH 102 may utilize other methods of characterrecognition understood by those skilled in the art. In one embodimentthe user may browse the website presented on the secondary display usingthe remote control 118.

Referring to FIG. 12, the HES 100 may contain a number of identical ornearly identical display devices 112 arranged to provide the user 120with a wide angle video experience. In one embodiment, the displaydevices 112 may be arranged to completely encircle the user 120providing a surround video experience. A source device 122 provides aprogram containing multiple video channels which, when displayed on themultiple display devices 112, provide a panoramic view of the programcontent to the user 120. The WHEH 102 may direct the source device totransmit each of the video channels to the appropriate display device.Alternately, the WHEH 102 may receive the transmission from the sourcedevice, and transmit the appropriate video channel to the correspondingdisplay device 112. In an alternate embodiment, multi-channel videoprogramming may be displayed on a single display device 112 bycompositing the video channels by the WHEH 102 for display on a singledevice. For example, if a programming channel contains data for threeseparate video channels that can be displayed to form a multi-displayprogram, and the HES 100 contains only one display device, the WHEH 102can composite the three video channels to be displayed on the onedisplay device 112 in the HES 100.

Systems using a WHEH 102 can be supplied in a low-security configurationto ease installation by non-technical users. In an embodiment withhigher security, the user 120 can enter a code on one device and confirmthe code on another device or on the user interface. In anotherembodiment, various system components can ship with awareness of uniqueidentifiers of other devices in the system.

The WHEH 102 may be used to wirelessly connect musical devices. Musicalsource devices and musical sink devices can be connected to a mixingboard containing an advanced embodiment of the WHEH 102. Musical sourcedevices include, but are not limited to, musical instruments,microphones, effects systems, and amplifiers. Musical sink devicesinclude by are not limited to speakers, and audio monitors. The mixingboard acts as both a sink unit and source unit, and is a convenientlocation for placement of the WHEH 102. The musical devices are allregistered with the WHEH 102 similar to the produce procedure describedabove for the HES 100.

The WHEH 102 may detect the specific user or users of the HES 100through RFID, image capture and analysis, or other personalidentification technologies understood by those skilled in the art. Inone embodiment, the remote control 118 may be equipped with afingerprint scanner used for identification of the user 120. Theidentification of the user 120 can be used to control access to variousdevices of the HES 100 based on a set of rules customizable for eachuser of the HES 100 by an authorized user (i.e., parent). For example,access to one or more devices can be denied based on time of day.Similarly, access to certain programming channels can also be deniedbased on a television program rating system, time of day, or selectedchannels. For example, users identified as children may not be allowedaccess to an Xbox® gaming console before 5 PM on weekdays or may not beallowed to view channels showing television programming rated TV-MA(under the US TV Parental Guidelines).

Characteristics of the HES 100 may be automatically adjusted based onidentification of the user 120 by the WHEH 102. Characteristics of theHES 100 include physical characteristics, such as the height ororientation (e.g. rotation, tilt) of the display device or speakers, andsystem characteristics, such as volume or equalization of the audio, orchannel on the STB 106. A profile may be stored in the WHEH 102 withinformation about the characteristics of the HES 100 corresponding theuser 120. The profile may be set by the user 120 or the WHEH 102 maystore the last configuration of the HES 100 for each user 120 and returnthe HES 100 to that configuration when the user 120 begins anentertainment session. In one embodiment, the profile also containsinformation about positioning or adjustments of viewing location 121,such as a couch or chair equipped with automatic adjustment mechanismsunderstood by those skilled in the art.

The embodiments of the present invention may be implemented with anycombination of hardware and software. If implemented as acomputer-implemented apparatus, the present invention is implementedusing means for performing all of the steps and functions describedabove.

The embodiments of the present invention can be included in an articleof manufacture (e.g., one or more computer program products) having, forinstance, computer useable media. The media has embodied therein, forinstance, computer readable program code means for providing andfacilitating the mechanisms of the present invention. The article ofmanufacture can be included as part of a computer system or soldseparately.

While specific embodiments have been described in detail in theforegoing detailed description and illustrated in the accompanyingdrawings, it will be appreciated by those skilled in the art thatvarious modifications and alternatives to those details could bedeveloped in light of the overall teachings of the disclosure and thebroad inventive concepts thereof. It is understood, therefore, that thescope of the present invention is not limited to the particular examplesand implementations disclosed herein, but is intended to covermodifications within the spirit and scope thereof as defined by theappended claims and any and all equivalents thereof.

What is claimed is:
 1. A control device comprising: a processor; atransmitter electrically coupled to the processor; a sound detectiondevice electrically coupled to the processor; the processor configuredto: transmit, wirelessly via the transmitter, information to a wirelesshub to cause control of a display device, in a first zone, registered tothe wireless hub, wherein the control device communicates with thewireless hub using orthogonal frequency division multiplexing (OFDM) andBluetooth; receive, from the wireless hub based on a user initiatedsource change, a control signal to assign one of a plurality of buttonsfor control of the display device; the transmitter is configured totransmit to the wireless hub a first signal, based on a voice commanddetected by the sound detection device, to cause control of content anda second signal based on a gesture command, detected by a motion sensingcomponent, to cause a source change of the display device and to causecontrol of another display device in a second zone; and wherein themotion sensing component detects one of a predetermined set of physicalmotions of the control device.
 2. The control device of claim 1, whereinthe wireless hub is in stand-by prior to transmission of the firstsignal.
 3. The control device of claim 1 further comprising: theprocessor is configured to send a command to substantiallysimultaneously display a first video output on the display device and todisplay a second video output on the another display device.
 4. Thecontrol device of claim 3, wherein the first video output and the secondvideo output are different.
 5. The control device of claim 3, whereinthe second video output includes complimentary content to the firstvideo output.
 6. The control device of claim 3, wherein the second videooutput includes an alternate view of the first video output.
 7. A methodperformed by a control device, the method comprising: transmitting, to awireless hub, information wirelessly to control a display device, in afirst zone, registered to the wireless hub, wherein the control devicecommunicates with the wireless hub using orthogonal frequency divisionmultiplexing (OFDM) and Bluetooth; receiving, from the wireless hubbased on a user initiated source change, a control signal to assign oneof a plurality of buttons for control of the display device;transmitting, to the wireless hub based on a voice command, a firstsignal to control content and a second signal, based on a gesturecommand, for a source change for the display device, wherein the secondsignal controls another display device in a second zone; and wherein thegesture command is one of a predetermined set of physical motions of thecontrol device.
 8. The method of claim 7, wherein the wireless hub is instand-by prior to transmission of the first signal.
 9. The method ofclaim 7 further comprising: sending a command to substantiallysimultaneously display a first video output on the display device and todisplay a second video output on the another display device.
 10. Themethod of claim 9, wherein the first video output and the second videooutput are different.
 11. The method of claim 9, wherein the secondvideo output includes complimentary content to the first video output.12. The method of claim 9, wherein the second video output includes analternate view of the first video output.